Guided drilldown framework for computer-implemented task definition

ABSTRACT

Techniques and solutions are described for configuring a computer-implemented process defined by a data model. The data model includes a plurality of data objects, each data object having an object type. Displays are rendered that request selection of first and second values for respective first and second data objects of first and second object types. The first and second values are assigned to the respective first and second data objects. The computer-implemented process defined by the data model is executed, using the first and second values, to provide execution results.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/428,980, filed Jun. 1, 2019, which is hereby incorporated herein byreference.

FIELD

The present disclosure generally relates to configuringcomputer-implemented processes to guide a user through a task.Particular embodiments are described with respect to user interfacedisplays that guide a user in configuring and executing a machinelearning task.

BACKGROUND

Many computer-implemented processes can be very complex for users,particularly for new users or users who are operating in an area that isoutside of their core area of expertise. Inefficient methods of guidinga user through a task can increase the time needed for the user tocomplete the task, as well contributing to the user's frustration.Particularly if the task is somewhat optional, a user may becomefrustrated to the point that they do not complete the task, or revert toa less efficient way of accomplishing the task. Task complexity can alsoincrease the chances for task errors to occur, or for the task to becompleted inefficiently using a computer. For example, if the taskinvolves configuration settings or other parameters that may affectcomputer performance, user inexperience and frustration can result inexcess computer resource use, such as processor use, execution time,memory resources, or network resources. Accordingly, room forimprovement exists.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Techniques and solutions are described for configuring acomputer-implemented process defined by a data model. The data modelincludes a plurality of data objects, each data object having an objecttype. Displays are rendered that request selection of first and secondvalues for respective first and second data objects of first and secondobject types. The first and second values are assigned to the respectivefirst and second data objects. The computer-implemented process definedby the data model is executed, using the first and second values, toprovide execution results.

In one embodiment, a method is provided for configuring and executing acomputer-implemented process defined by a data model. A data model isreceived that includes a plurality of data objects. Each data object hasan object type. A first display is rendered, requesting selection of afirst value for a first data object having a first data object type. Thefirst display also provides a schematic diagram of the plurality of dataobjects in the data model. The first value is assigned to the first dataobject. A second display is rendered requesting selection of a secondvalue for a second data object having a second data object type. Thesecond display also provides the schematic diagram. The second value isassigned to the second data object. A computer-implemented processdefined by the data model is executed using the first value and thesecond value to provide execution results. The execution results aredisplayed to a user.

In another aspect, another method is provided for configuring andexecuting a computer-implemented process defined by a data model. Aselection of a data model type is received. The data model type isdefined by a sequenced or hierarchical relationship of a plurality ofdata objects of a plurality of different data object types. A data modelcorresponding to the data model type is retrieved. A first display isrendered requesting at least a first value to be assigned to a firstdata object of the plurality of data objects. The first data object hasa first type. The first display depicts the sequenced or hierarchicalrelationship of the plurality of data objects. A second display isrendered. The second display requests at least a second value to beassigned to a second data object of the plurality of data objects. Thesecond data object has a second type. The second display depicts thesequenced or hierarchical relationship of the plurality of data objects.A computer-implemented process defined by the data model is executedusing the at least a first value and the at least a second value.Execution results are displayed.

In a further aspect, a method is provided for configuring and executinga machine learning model. A selection of a task level is received. Aplurality of machine learning models, stored in a repository, aredetermined that satisfy the selected task level. A selection of amachine learning problem type or a use case is received. At least one ofthe plurality of machine learning models that satisfies the problem typeor the use case is determined. A representation of the at least onemachine learning model is displayed. The representation illustrates asequenced or hierarchical relationship between a plurality of componentsof the at least one machine learning model. At least a first componentof the plurality of components represents a data set to be processed andat least a second component of the plurality of components represents atleast one algorithm to be used to process the data set. Input specifyingthe data set is received. The specified data set is processed using analgorithm of the subset. Execution results are provided.

The present disclosure also includes computing systems and tangible,non-transitory computer readable storage media configured to carry out,or including instructions for carrying out, an above-described method(or operations). As described herein, a variety of other features andadvantages can be incorporated into the technologies as desired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example computing environment havinga guidance framework in which disclosed embodiments can be implemented.

FIG. 2 is a diagram illustrating an example of an algorithm library.

FIG. 3 is a schematic diagram illustrating hierarchical relationshipsbetween components of algorithms and other components of a data model.

FIG. 4 presents example pseudocode for various components of a machinelearning model.

FIGS. 5-9 are example user interface screens useable to select,configure, and execute a machine learning model.

FIGS. 10 and 11 are flowcharts of example methods for configuring andexecuting a computer-implemented task represented by a data model.

FIG. 12 is a flowchart of an example method for configuring andexecuting a machine learning model.

FIG. 13 is a diagram of an example computing system in which somedescribed embodiments can be implemented.

FIG. 14 is an example cloud computing environment that can be used inconjunction with the technologies described herein.

DETAILED DESCRIPTION Example 1 Overview

Many computer-implemented processes can be very complex for users,particularly for new users or users who are operating in an area that isoutside of their core area of expertise. Inefficient methods of guidinga user through a task can increase the time needed for the user tocomplete the task, as well contributing to the user's frustration.Particularly if the task is somewhat optional, a user may becomefrustrated to the point that they do not complete the task, or revert toa less efficient way of accomplishing the task. Task complexity can alsoincrease the chances for task errors to occur, or for the task to becompleted inefficiently using a computer. For example, if the taskinvolves configuration settings or other parameters that may affectcomputer performance, user inexperience and frustration can result inexcess computer resource use, such as processor use, execution time,memory resources, or network resources. Accordingly, room forimprovement exists.

As an example, machine learning is a topic that is of increasinginterest to users who are not computer scientists, or even programmersor software developers. Some applications try and facilitate the use ofmachine learning techniques by creating libraries of machine learningalgorithms, or by providing software that uses machine learning for avery specific application, such as an application that uses predefineddata sets (e.g., a particular collection of database tables) to provideset output using a set of pre-defined parameters.

However, such software applications may be insufficient for manypurposes. For example, it may be impractical for developers to providepredefined applications for every possible use case that may be ofinterest, particularly if users do not have data that is maintained in astandard format or has standard characteristics. Even for predefinedtasks, if a user does not understand what they are doing, and where theyare in an overall process, they may become frustrated. For many users,using machine learning techniques can be very beneficial, but may not beroutine or required. Accordingly, if the user does not understand whatthey are doing, why they are doing it, or how many steps they have left,they may become frustrated, and possibly give up. Further, whilepresenting steps to a user one step at a time may be useful in guidingthe user through a task, the user may be less likely to understand whatthey are doing, which may not facilitate their accomplishing new or morecomplex tasks, or trying variations for a task.

The present disclosure provides a guided drill down framework that canguide a user through a task that includes multiple steps and components.The guided drill down framework can include user interface displays thatdisplay options for a particular step or component in a process ormodel, but simultaneously also display a representation of the overalltask or model configuration process, such as showing a representation ofeach task or model component (which can correspond to separate userinterface screens which will be presented to a user during taskdefinition). In this way, a user can better understand how a currentuser interface display, and configuration of a particular task orcomponent, relates to an overall task. This understanding can facilitatethe user in completing the task and understanding task parameters, whichcan help the user complete similar tasks in the future, modify the task,or generate or try new, possibly more complex, tasks.

In many cases, various parameters of a task may be mutually exclusive.Task steps or model components may be sequential, or have a hierarchicalrelationship. Accordingly, configuration of one task or model parametermay limit available options or values that can be used for other task ormodel parameters. The guided drill down framework can guide a user incompleting the task by automatically restricting a user's selections fortask parameters based on previously supplied user input. Or, theframework can determine whether a user's selection is consistent withprior selections. In the event a user makes a selection that isincompatible with a prior selection, a warning can be provided to auser, and optionally a suggestion provided of acceptable values.

At least some cases, a guided drill down framework can guide a userthrough task configuration or completion based on sequential orhierarchical task steps. That is, the user can be presented with userinterface displays to complete task steps or configure components inorder of the sequence or level of the hierarchy. In many cases, a usermay be most confident in making selections for earlier task steps orcomponents at a higher level of a hierarchy. By guiding a user throughthe task in a “drill down” manner, and presenting acceptable options orvalues to a user, the user can understand how later, or deeperhierarchical, steps or components relate to earlier steps or components,or steps or components at a higher hierarchical level.

Even with a guided drill down framework, many tasks may be so complex,with so many options, as to still be overwhelming to novice users. Ifsoftware applications cater to users of one particular experience level,it can disadvantage other types of users. For example, software directedto a novice user may omit options that may be desired by a moresophisticated user. Accordingly, a disclosed guided drill down frameworkmay adapt based on a user's experience level, or on a task level (i.e.,a task complexity level, such as “advanced,” “intermediate,” “novice,”etc.), which level can be explicitly set by or for a user, or which maybe inferred through user input (for example, a user failing to selectoptions or values for particular parameters, or selecting a “default”option). The options at one or more steps in a task, typically in aplurality of steps or all steps, can be tailored based on the experienceor complexity level.

Disclosed technologies can find particular use in the field of machinelearning. Even among computer scientists, developers, and programmers,machine learning can be a complicated and specialized field that may notbe well understood outside of those who specialize in the area. Machinelearning techniques are increasingly being applied to “practical”problems that may be of interest to enterprise users. However, outsideof specially designed, special-purpose programs, it may be difficult forenterprise users to apply machine learning to problems that they face.

In the case of machine learning, a machine learning task may be definedby a process model, where the process model can have many steps orparameters, which can be represented by objects having various objecttypes. Selection of a value or option for one step, or for oneparameter, may limit options or values that can be selected for othersteps or parameters. As an example, if a user selects a machine learningtask relating to “classification,” that may limit a set of algorithmsthat may be used with the task, data sets that can be used with thetask, output views that are compatible with results produced by thetask, etc. If a particular algorithm is selected, it may further limitwhat data sets can be used with the algorithm, other data processingthat can be performed (e.g., sending the results of a first machinelearning algorithm for processing by a second machine learningalgorithm), or particular visualizations or output screens that can beused to present, and optionally interact with, results of the machinelearning task.

Different process models can be created for users having differentexperience levels or for different complexity levels, or particularelements of a process model can be classified by user experience orcomplexity level such that a given process model can practically be usedto define particular implementations of the process model for a givenuser experience or complexity level. Generally, more options areprovided to users having greater levels of sophistication. For example,novice users may be given a more limited type of machine learning taskto accomplish (for example, classification, but not clustering), mayhave the tasks presented in different manners (for example, rather than“classification,” a task may be defined with respect to a particulartype of data or result to be provided, such as classifying individualsas productive or unproductive). Particularly when provided with otherdisclosed features, such as a guided drill down approach, providingprocess models for different experience levels can help a user learn atask such that the user can progress to higher levels of taskcomplexity.

Disclosed technologies can provide various advantages. One or more ofusing a guided approach, a drill down approach, or an approach thatprovides different process models for users have different experiencelevels, or for different complexity levels, can facilitate a userlearning how a task is solved, which can allow them to configure tasksmore efficiently or progress to more complex tasks. By reducing userfrustration, these approaches can also improve the user's experience intask definition and execution, and help to improve the level of use oftask definition and execution tools, and improving user completion rates(that is, reducing the number of users who fail to complete a taskdefinition).

Process models can also reduce programming effort, and also storagerequirements, since a given process model can be implemented/configuredin multiple ways. So, rather than having different models for each setof task parameters, a single process model can be dynamically configuredas needed for a given situation. In cases where less sophisticated usersmay be configuring a task, having process models that have some steps orparameters configured with default values, or not being changeable (orperhaps even visible to) an end user can help avoid configuration errorsor configurations which might be less efficient. Thus, disclosedtechnologies can improve computer resource use, such as having a taskuse less CPU cycles, memory, or network resources.

Example 2 Example Computing Environment With Guidance Framework

FIG. 1 illustrates components in an example computing environment 100 inwhich disclosed technologies can be implemented. At a high level, thecomputing environment 100 includes a guidance framework 108 that can beused to create and execute process models, such as process models in theform of machine learning models 128. The computing environment 100 alsoincludes a machine learning framework 158 that can be called to executeinstances of machine learning models 128, and one or more data sources,such as a database 170.

The guidance framework 108 can include a configuration component 112.The configuration component 112 can be used to create instances of amachine learning model 128, such as choosing particular options for aparticular model, or particular values for model parameters, as will befurther described. In some cases, the configuration component 112 canalso be used to define or edit machine learning models 128.

The guidance framework 108 can include an executor 116. The executor 116can manage execution of a particular instance of a machine learningmodel 128, such as after a user has configured the model using theconfiguration component 112. The executor 116 may access the machinelearning framework 158 in order to execute an instance of a machinelearning model 128, such as by calling an API 162. In some cases, theexecutor 116 may be configured to access multiple machine learningframeworks 158, such as by calling an API 162 of a machine learningframework 158 desired to be used by a user, or as specified in aninstance of a machine learning model 128.

The guidance framework 108 can include a user interface 120. The userinterface 120 may include a plurality of user interface screensconfigured to guide a user through a configuration process using theconfiguration component 112. The user interface 120 may also provideuser interface screens to allow a user to execute a machine learningmodel, to view the status of a machine learning task being executed, orto view execution results.

The machine learning model 128 includes various objects, which can alsobe referred to as component or entities, that define the machinelearning model, including specifying how a machine learning task will beconducted, and a plurality of sub-components or sub-entities, which alsocan be objects, to be configured by a user for a particular task. Theobjects can be associated with specific and discrete types, such as adata set type, an algorithm type, or an output settings type. Themachine learning models 128 can be stored in a repository 124. As willbe further described, a user may browse at least a portion of therepository 124, select a particular machine learning model 128 for use,and then configure and execute the model using the configurationcomponent 112 and the executor 116.

The components of (e.g., represented by objects in) the machine learningmodel 128 can be categorized according to the different functions theyserve. For example, the components can be categorized into inputs,outputs, and processing components that read inputs, perform processing,and provide outputs. A given object or component may have one or moresubobjects or subcomponents, as will be further described.

A machine learning model 128 can include a metadata object type 132. Themetadata object 132 can describe various features of the machinelearning model, such as one or more types associated with the machinelearning model. A type can be a type of problem the model is intended tobe used with, such as whether the model relates to a classificationproblem or a clustering problem. The type can also refer to whether themachine learning model 128 is associated with one or more generalmachine learning techniques, such as supervised learning, unsupervisedlearning, semi-supervised learning, or reinforcement learning.

The metadata object 132 can include identifiers for various use-caseswhere a machine learning model 128 can be useful, such as imagerecognition, text recognition, or particular elements of enterpriseresource planning software, such as demand forecasting. The metadataobject 132 may include identifiers for particular types of users forwhich the machine learning model 128 may be useful, such as a userhaving a particular title or role. That is, for example, a given machinelearning model 128 may generally be useful for manufacturing orproduction, but not as useful for distribution. The metadata object 132can also include an identifier indicating the relative complexity levelof a given machine learning model 128 relative to other machine learningmodels (i.e., can provide an ordering or ranking). Examples of differentidentifiers can be “high, medium, low,” “expert, intermediate, novice,”or “1, 2, 3.” In some cases, an identifier can be tied to other metadatacomponents. For example, a particular machine learning model may be“easy” if used by a user having a particular role, but may be “hard” ifused by a user having a different role.

The metadata object 132 can serve additional purposes, such as todescribe the components used in the machine learning model, the numberof such components in the model, or the relationship between thecomponents.

The machine learning model 128 can include one or more data set objectsof a data set object type 134. A data set object type 134 can includemetadata 136 that describes features that should be associated with datasets that are useable with a given machine learning model 128. Forexample, the metadata 136 can specify a particular type for the dataset, such as a format type (e.g., XML, JSON, CSV, tabular), or a typeassociated with a particular software application (e.g., a file type).The metadata 136 can specify a minimum or maximum number of data items(e.g., a number of records). The particular properties can includeadditional features, such as requiring a particular attribute be presentin the data set or an attribute that can be mapped to a particularelement of another component of a machine learning model 128. Forexample, the metadata 136 can specify that a particular attribute of thedata set be specified as a label for corresponding data values.

The metadata 136 can specify how a data set object 134 interacts withother components of a machine learning model 128, such as with one ormore algorithm objects 140 or one or more output settings objects 148(e.g., that define output settings or parameters for the machinelearning model) included in the machine learning model. For example,metadata 136 can specify what kind of algorithm objects 140 or outputsettings objects 148 can be used with a particular data set object 134.

During user configuration of an instance of the machine learning model128, the user can assign one or more data sources to the data set 134.For example, a user may select a particular file, file path, URI, orother indicator of a data source to be used. In many instances, the datasource will include data stored in a database. In this scenario, a usercan specify particular data to be used using a query language statement,such as a query language statement to retrieve data to be used in thedata set object 134. Optionally, the user can specify how data from adata source will be processed or selected for use in the data set object134, such as specifying filter criteria, formatting operations,aggregation operations, or calculations to be performed using data froma data source, where the results are to be included in the data setobject. In at least some cases, this processing can be specified as partof a query language statement that retrieves data from a database.

The machine learning model 128 can include one or more algorithmsassociated with an algorithm object type 140. An algorithm object type140 can be associated with metadata 144. The metadata 144 can specifywhat types of algorithm objects 140 can be used in a particular machinelearning model 128, including based on particular data set objects 134or output settings objects 148 available for the machine learning model,generally, or data set objects or output settings objects that have beenselected for a particular instance of the machine learning model 128.That is, like other components of the machine learning model 128, aparticular machine learning model may restrict what data sets 134 (ordata set properties) can be used with the model, what algorithm objects140, or values or settings for such objects, can be used with the model,and what output settings objects 148, or values or settings for suchobjects, can be used with the model. Selections made for a giveninstance of a machine learning model 128 may further restrict whatalgorithm objects 140 can be used. For instances, a given machinelearning model 128 may have already had one or more data set objects 134selected (or configured), or one or more output settings objects 148selected (or configured). Some of the algorithms that otherwise may havebeen useable in an instance of the data model 128 may not be compatiblewith such selections, and the metadata 144 can indicate dependenciesbetween an algorithm object 140 and data set information or informationregarding output parameters.

An algorithm object 140 can be associated with one or more objects of asetting information object type 142. A setting information object 142can represent a subcomponent or subentity of a machine learning model128. Each algorithm object 140 can be associated with its own settinginformation. Thus, the selection of an algorithm object 140 dictates atleast in part what setting object 142 may be available, or how they canbe configured. The setting information object 142 can also depend onother factors, such as a level (e.g., “expert” or “novice”) associatedwith the machine learning model 128, or other information. In somecases, setting information objects 142 may be set for a given algorithm140 object, and made visible to a user, but not changeable by a user.Or, all or a portion of the setting information 142 can be made visibleto the user, and at least a portion of the setting information can beset or altered by a user. For example, some elements of the settinginformation object 142 can represent default values. Setting informationobjects 142 can be restricted or set based on other factors, such as aparticular use case selected by a user, a particular data set object 134selected for use, or particular output parameters of an output settingsobject type 148 selected for use.

Output settings specified in an instance of an output setting objecttype 148 can influence what data is output by an algorithm object 140,how the data is formatted, how the data is interpreted, or how the datais displayed. Data output by an algorithm object 140 can includeresults, but can also include information regarding the operation orperformance of the algorithm, such as confidence values or weightingvalues determined during execution of the algorithm, as well asinformation regarding the efficiency or computer resource use of thealgorithm.

Specifically, the output settings objects 148 can include interpretationsettings in an interpretation settings object type 150. Theinterpretation settings used in an instance of the interpretationsettings object type 150 can be based on a particular application ordata set object 134 used by the algorithm. For example, data regarding amanufacturing process may be interpreted differently than data relatingto distribution of goods. The interpretation setting objects 150 can,among other things, specify particular user interface screens or userinterface elements that will be presented to a user to help the userinterpret the results of an algorithm associated with an algorithmobject 140.

The output settings objects 148 can also include results settings in aninstance of a result settings object type 152. Results settings in aresult settings object type 152 can indicate what results of analgorithm in an algorithm object 140 should be output, and how theoutput should be formatted or displayed. Results output in accordancewith the result setting objects 152 can be related to an interpretationprovided by the interpretation setting objects 150. However, in general,the results setting objects 152 describe “raw” results provided by thealgorithm object 140, as compared with the interpretation settingobjects 150, which influence how the results are explained to the user.

The output settings objects 148 can include performance settings in aperformance settings object type 154. The performance settings objectcan be used to specify output to be provided to a user regarding theperformance of an algorithm object 140 or the performance of one or morecomputing devices while executing the algorithm. Algorithm performanceinformation can include error statistics or confidence statistics.Resource information can include a total algorithm execution time, anumber of iterations performed, or information regarding CPU, memory, ornetwork use.

Although shown as including a single instance, at least some components(or sub-components) of a machine learning model 128 may have multipleinstances. For example, multiple data set objects 134 can be accessed bya single instance of a machine learning model 128. Or, one or more dataset objects 134 may be analyzed using multiple algorithm objects 140.For example, results of a first algorithm object 140 may be furtheranalyzed using a second algorithm. In the event that there are multipledata set objects 134 or multiple algorithm objects 140 used, it may beuseful to have multiple instances of the output settings objects 148.

The computing environment 100 can include components for use in creatinga machine learning model 128, instantiating a machine learning model, orexecuting a machine learning model. For example, a machine learningmodel 128 can be accessed by a machine learning framework 158. Themachine learning framework 158 can include, for example, algorithms 160specified by the algorithm setting objects 140 of a machine learningmodel 128. When an instance of a machine learning model 128 is executed,algorithms 160 specified in the instance can be called to process dataspecified by a data set object 134. The machine learning framework 158can include an API 162 that includes methods for accessing thealgorithms 160, such as for initiating processing of a data set object134 using an algorithm, setting algorithm parameters, monitoringalgorithm execution, or retrieving execution results.

The machine learning framework 158 can include a resource monitor 164.The resource monitor 164 can monitor computer resource use duringexecution of an instance of a machine learning model 128, including sothat performance results can be provided as specified by the performancesettings 154. The machine learning framework 158 can include componentsto assist in formatting, displaying, or interpreting results of analgorithm 160. These components can include visualization templates 166.The visualization templates 166 can include templates for visualizationssuch as scatter plots, bar charts, area charts, line charts, histograms,pie charts, tree maps, fishbone diagrams, or matrix diagrams. Thevisualization templates 166 can be associated with metadata indicatingparticular types of algorithm objects 140 or categories of machinelearning tasks with which a particular visualization template might beused. The API 162 can include methods for calling a visualizationtemplate 166 and generating a visualization using a template. The API162 for visualization can be used by external components, such as theoutput settings objects 148 of the machine learning model 128, but alsoby other components of the machine learning framework 158.

The machine learning framework 158 can include user interface controls168, which in at least some cases can include all or a portion of anentire user interface screen. User interface controls 168 may becomponents that facilitate configuring or executing an algorithm 160,selecting or interacting with a visualization template 166 (or avisualization generated using such a template), or for viewinginformation associated with the resource monitor 164.

Although a single machine learning framework 158 is shown, in someaspects, a computing environment 100 can include multiple machinelearning frameworks, which frameworks may be configured as shown for theframework 158, or may be configured differently in whole or part. In atleast some cases, a machine learning model 128 can be at least partiallydecoupled from a particular machine learning framework 158. For example,a machine learning model 128 can be executed using different machinelearning frameworks 158 by accessing the respective API of suchframeworks.

The computing environment 100 can include one or more data sources, suchas data sources that hold data specified by a data set object 134 of aninstance of a machine learning model 128. For example, the computerenvironment 100 includes a data source in the form of a database 170.The database 170 can store various objects, such as a plurality ofrelational database tables 172. One or more views 174 may be definedwith respect to the tables, as may one or more cubes (e.g., OLAP cubes).In some cases, a cube 176 can exist as an instantiated object withmaterialized data, while in other cases a cube can define a schema thatis in turn defined with respect to the tables 172 or the views 174. Thedatabase 170 can include additional features, such as one or more storedprocedures 178. In some cases, a stored procedure 178 can be used aspart of pre-processing a data set specified in a data set object 134prior to the data set being processed using a machine learning algorithm160. Or, a stored procedure 178 may be called to process final orintermediate results produced by an algorithm.

Although the computing environment 100 is described with respect to amachine learning model 128 as the process model, elements of thecomputing environment 100 can be adapted for use with other types ofprocess models. Generally, the components shown in the computingenvironment 100 can be adapted to use such different types ofprocessing. For example, various process models for a particular type ofprocess may have a structure similar to that of the machine learningmodel 128, where objects for various component or entities that shouldbe defined for the process model are included in the template. Thesecomponents or entities can specify permitted values or options,including accounting for dependencies between components or entities,such that a selection made for one component or entity may restrictoptions that can be selected for, or values assigned to, othercomponents or entities. Rather than a machine learning framework 158,another type of process execution framework can be accessed in order tocarry out a process defined by a particular instance of a process model.

Example 3 Example Algorithm Library

FIG. 2 illustrates an example of an algorithm library 200. The algorithmlibrary 200 can be an example of algorithm objects 140 that can beselected (or available for selection) in a machine learning model 128 ofFIG. 1 . Similarly, the algorithm library 200 can be an example ofalgorithms 160 available in a machine learning framework 158, which cancorrespond to the algorithms in the algorithm objects 140.

The algorithm library 200 can include various categories 212 ofalgorithms, shown as categories 212 a-212 d. For example, the algorithmlibrary 200 is shown with categories corresponding to supervisedtechniques 212 a, unsupervised techniques 212 b, semi-supervisedtechniques 212 c, and reinforcement learning 212 d. Each category 212can be associated with metadata 216. The metadata 216 can include a usecase component (or object) 220, which includes identifiers for varioususe cases where an algorithm in the category 212 may be useful. Forexample, a use case 220 may be “manufacturing,” or “manufacturing faultanalysis.”

The metadata 216 can include indications, such as in an indicationsobject type 222, where an indication may include requirements orconditions. For example, a given category 212 may require a particulartype of data, such as labelled data or numerical data. Or, a category212 may require a set of training data before algorithms can be usedwith a set of data to be analyzed. A guidance framework, such as theguidance framework 108 of FIG. 1 , can use the indications 222 to helpdetermine what selections can be made when instantiating a machinelearning model 128.

The indications 222, or other elements of the metadata 216, can specifywhat category, user types, etc. for which algorithms in the category 212may be available, or for which they may be most useful. For example, onecategory 212 may have indications 222 specifying that the category hasalgorithms that are suitable for novice users, intermediate users, andexpert users. Another category 212 may have indications 222 specifyingthat the category only has algorithms suitable for intermediate usersand expert users.

The metadata 216 can include problem type metadata, such as in a problemtype object type 224. Problems types can include types such as“classification” or “clustering.” The problem type object 224 may beused by the guidance framework 108 to help determine what selections foran instance of a machine learning model 128 are displayed as availableoptions, or whether selections are valid. The metadata 216 can includeadditional elements. For example, the metadata 216 can include anelement describing the category 212 to which it applies.

Each category 212 of algorithms can include one or more algorithms 228,shown as algorithms 228 a-228 h. An algorithm 228 generally includesmetadata 230 (which can be an object for metadata for algorithms) andone or more settings 232 (which can be maintained in an object of asettings object type). The metadata 230 can be similar to the metadata216, including having elements (not shown) that describe use cases,indications, and problem types associated with an algorithm 228. Inparticular, the metadata 230 can specify what type of users may access aparticular algorithm 228, such as whether the algorithm 228 is suitablefor use by novice, intermediate, or expert users. The metadata 230 mayalso specify settings 232 that are available for selection orconfiguration for different levels of users, or supply default valuesfor a setting. In other cases, the settings 232 may themselves havemetadata, which can specify categories of users for which the settingwill be displayed, or default values.

When a machine learning model is defined, such as a machine learningmodel 128 of FIG. 1 , the template for the model can specify whichcategory 212 the model is defined for, or can specify use cases,indications, or problem types that can be compared with metadata 216 ofthe different categories 212 to help determine which categories mayapply to a given model. Similarly, the machine learning model 128 candirectly specify one or more algorithms 228, or information provided forthe model may be compared with the metadata 230 to determine whichalgorithms may be appropriate for use with the model. When a machinelearning model is processed using the configuration component 112, auser may be prompted to select a category 212, or to provide informationuseful to determine one or more categories, from which the user may thenselect. Once the category 212 is selected, the machine learning model,the selected category, and any other information regarding the user oruse case may be used to display available algorithms to a user, wherethe user can select an algorithm to be used with a particular instanceof the machine learning model. Any settings 232 which are required to beconfigured by a user, or which the user has the option to change, can beconveyed to the user. Once a user has configured an algorithm 228, andselected other components of a machine learning model, such as the dataset object 134 and the output settings objects 148, the algorithm 228can be executed and results provided to the user.

Example 4 Example Object Hierarchy

FIG. 3 is a diagram illustrating a hierarchy 300 that can be used todescribe elements of a machine learning model. In particular, thehierarchy 300 illustrates how selections made at one level of thehierarchy can affect choices available at other levels

The hierarchy 300 can include a root node 308, where a selection can bemade at a first level of the hierarchy of a particular class 314 ofalgorithms, shown as classes 314 a, 314 b, 314 c. Children of a class314, at a second level of the hierarchy 300, can represent particulartypes 318 of algorithms (shown as types 318 a, 318 b) in thecorresponding class 314. Note that selection of either a type 318 or aclass 314 restricts other available options, even if the hierarchy 300is not entered (e.g., a machine learning model is not configured from)the root node 308. That is, selecting the “supervised” class 314 arestricts the available types to “classification” 318 a and “regression”318 b. Correspondingly, in some cases, “regression” 318 b might beselected initially, thus setting a traversal through the hierarchy 300such that only options satisfying “regression” 318 b and “supervised”314 a are available or valid.

The children of the second level of the hierarchy 300, at a third level,can correspond to particular algorithms 320, shown as algorithms 320a-320 h. As before, selections made at a higher level in the hierarchy300 restrict selections available at the third level. For example, if“classification” 318 a is selected, then at most algorithms 320 a-320 ewould be available. That is, as has been described, the experience levelof a user, or a similar parameter, may be used to further restrictalgorithms 320 that are made available, as will any selections regardinga use case or data set for use with a particular instance of a machinelearning model.

The final, fourth level of the hierarchy 300 corresponds to settings 322of a particular algorithm 320 a. For convenient presentation, onlysettings 322 a-322 d of algorithm 320 a are shown. Some or all of thesettings 322 may be made available to a user or displayed to a user.Similarly, in some cases, default values may be provided for some of allof the settings 322. Generally, as a user becomes more experienced, theycan choose machine learning models that makes a greater number ofsettings 322 available for a user to modify.

FIG. 3 also illustrates how different component or object types of amachine learning model can be related. For example, as described above,a particular data set may be useable with some machine learningalgorithms, but not others, and may be associated with metadatadescribing algorithms with which the data set can be used. A metadatatag 330 can be provided representing labelled data, and can beassociated or linked with supervised algorithms 314 a andsemi-supervised algorithms 314 b. Correspondingly, a metadata tag (notshown) for unlabeled data would not be linked to the supervisedalgorithm category 314 a. Similarly, result or output parameters, ormetadata for such parameters can be associated with other object types,such as algorithms or data set characteristics. As shown, particularresult/output parameters 334 can be associated with a decision treesalgorithm 320 f.

As described above, objects in the hierarchy 300 can be associated witha complexity level. In at least some cases, a complexity level for aparticular instance of a machine learning model will retrieve objectsassociated with that complexity level or a lower complexity level. Forexample, an “intermediate” complexity level will retrieve “intermediate”objects as well as “novice” objects. However, the technology can beimplemented in a different way, such as a complexity level onlyretrieving objects of that complexity level. This can be useful, such asin cases where objects or settings for different complexity levels maybe inconsistent or incompatible.

Example 5 Example Code for Model Object Types

Components of a machine learning model can be stored in various types ofrepresentations, including as records in database tables, metadatadescriptions (including in XML, or JSON format), or as classes orsimilar abstract data types. FIG. 4 provides a pseudocode description ofvarious model components.

In particular, code fragment 408 represents an example class thatdefines an interface for an algorithm. That is, the interface definesproperties that any algorithm in the class must implement. The interfacefragment 408 includes a data member 412 for a name (e.g. “naive Bayes”),a data member 414 for a type (e.g., “supervised,” “unsupervised”), adata member 416 for a problem type (e.g., “classification,”“clustering”), a data member 418 for a user level (e.g., “novice,”“expert”), a data member 420 for an input type (e.g., “labelled,”“unlabeled,” “numeric,” “string”), a data member 422 for an output type(e.g., “numerical matrix,” “array of strings,” “naive Bayes,”“clustering”), and a data member 424 to actually hold output results ofan instance of the algorithm.

In the case of an abstract data type, the code fragment 408 can declaremethods that can be called to execute an instance of the algorithm. Forexample, a method 428 can be called to start an algorithm, a method 430can be called to stop execution of the algorithm, a method 432 can becalled to get a current status of the algorithm, and a method 434 can becalled to get the results of algorithm execution.

Code fragment 438 provides a declaration of a class for a particularalgorithm, random forests, that implements the interface of the codefragment. The code fragment 438 includes the data members 412, 414, 416,418, 420, 422, 424, but has values assigned to the variables that areappropriate for the random forests algorithm. For example, the datamember 412 has been assigned a value of “Random Forests,” while valuesfor data members 414, 416 indicate that the algorithm is a supervisedmachine learning algorithm that can be used to solve classificationproblems. The value for the data member 418 indicates that the algorithmis of an intermediate level. So, typically the random forests algorithmwould be available for use by intermediate and higher users (e.g.,intermediate and expert users), but not by users of lower thanintermediate level, such as novice users. Data members 420, 422 havebeen assigned values indicating that the algorithm requires labelleddata, and that the output of the algorithm is a tree data structure.

The code fragment 438 for the random forests class adds additional datamembers 440, 442. The data members 440 correspond to integer values forsettings associated with the random forests algorithm. The data members442 can specify a user level associated with a respective setting datamember 440, which can be used to determine which settings will bedisplayed to a user or available for modification by a user.

The methods 428, 430, 432, 434 are present in the code fragment 438, buthave been modified to provide specific implementations of the interfacemethods. In particular, the method 428 to start an instance of thealgorithm includes arguments of a job name or identifier 446 (e.g., foridentifying the job on a system where the algorithm is being executed),a data set identifier 448 (e.g., for identifying a data set to beretrieved, or a data structure that includes data to be operated on bythe algorithm), and the data members 440 corresponding to the algorithmsettings. The methods 430, 432, 434 are shown as having the job name 446as an argument.

Code fragment 452 represents a metadata definition of a data source. Thecode fragment 452 can be a JSON representation of a data source. Thecode fragment 452 includes key-value pairs for properties such as a dataset name 456, a data set type 458 (e.g., table, CSV file, spreadsheetfile), a flag 460 indicating whether the data set is labelled, datatypes 462 of columns included in the data set (if the code fragment 452is for a representation of a data set type that includes columns), anumber of columns 464 in the data set, and a number of records 466 inthe data set. Metadata definitions of data sources can be differentbased on the nature of the data source, and, more generally, can includemore, fewer, or different properties than shown. For example, a datasource definition can specify algorithms, or classes of algorithms foruse with the data source, or particular use case scenarios or problemtypes with which the data set may be used.

Example 6 Example User Interface Screens for Model Selection,Configuration, and Execution

FIGS. 5-9 are example user interface screens 500, 600, 700, 800, 900that can be present to a user to help guide the user through selecting,configuring, and executing a machine learning process, such as a processthat is based on a machine learning model 128 of FIG. 1 . The userinterface screen 500 can be a screen initially presented to a user todefine an instance of a machine learning problem, such as an instance ofa machine learning model 128. The user interface screen 500 can includea control 508 that allows a user to select whether to use a guidedanalysis, such as according to a disclosed technology, or whether, forexample, the user wishes to manually define a machine learning task. Ifthe user has set the control 508 to provide a guided analysis,additional user interface elements can be provided to allow the user toselect a particular machine learning task associated with a particularmachine learning model 128.

In particular, a user may enter a level, such as “beginner,”“intermediate,” or “expert” in a field 512. Alternatively, radiobuttons, a drop-down menu, or similar controls could be provided toallow a user to select a level. As explained, the level can be used torestrict what machine learning models 128 will be available to a user,or optionally components of models that will be available for userconfiguration. A user may be permitted to select a field of use in afield 514. For example, a field may be “manufacturing,” “distribution,”or another identifier of a particular field of use/use case scenariosthat are of interest to a user. In at least some cases, the fields ofuse available for selection for the field 514 can be limited based onthe level provided in the field 512. For example, each experience levelmay correspond to a definition, such as defined in a schema or anabstract data type, that specifies what fields of use are available forthat experience level.

A user may select a problem type in field 516. As shown, a popup menu518 can list available problem types. The available problem types can belimited at least in part based on other user selections, such as theexperience level provided in the field 512, the field of use provided infield 514, or based at least in part on both of these factors.

A preview and workflow panel 520 can provide visual representations 524of machine learning models that are available. The visualrepresentations 524 can be filtered or generated, at least in somecases, based on user selections made for the fields 512, 514, 516. Inother cases, the visual representations 524 can be provided even if notvalid under current user selections, but such visual representations canbe displayed in a manner to illustrate that they are currently not validselections based on the user input. For example, the visualrepresentations 524 that are incompatible with current selections can bedisplayed as grayed out.

The visual representations 524 can be helpful, as they provide anindication to a user of how complex a particular machine learning modelis, and accordingly how complex it will be, or how much time it willtake, to configure and execute a particular machine learning task. In atleast some cases, elements 526 of a visual representation 524 can depictparticular objects, components, or subcomponents of a machine learningmodel, and can represent interactions or relationships between suchobjects, components, or subcomponents. For example, elements 526 can beshaped, colored or otherwise shown in a manner to distinguish betweendata sets, algorithms, or output parameters. Similarly, subcomponentssuch as algorithm settings and different types of output parameters canbe visually distinguished. Displaying such information can assist a userin better understanding how various components of a machine learningmodel are used and relate to each other, which can make configuring thetask easier and less error prone. In addition, as the user is learningmore about machine learning, the user may be able to progress to morecomplex tasks (e.g., they may move from a beginner level to anintermediate level, or from an intermediate level to an advanced level).

Once the user has selected a particular machine learning model, such asthrough selections made through the user interface elements 512, 514,516, or, in at least some cases, selection of a visual representation524, the screen 500 can be replaced with the screen 600. The user mayalso be provided with navigation options to move to a different screen,or to help the user understand a workflow involved with configuring andexecuting a task, including how a current step/user interface screenrelates to other steps. In particular, the screen 500 can providenavigation links 530 that describe steps in the workflow and whichoptionally can be selected to move to a corresponding workflow step,such as to a user interface screen associated with the associatedworkflow step. As shown, navigation link 530 a is shown as highlighted(e.g., by underlining), indicating that it is the active step/the screen500 relates to this step.

The screen 600 of FIG. 6 can allow a user to select a data set to beprocessed with a machine learning task, such as being specified as adata set to be used with an instance of a machine learning model 128 ofFIG. 1 . The screen 600 can be presented to a user after the screen 500has been completed, or based on other input, such as by selecting theappropriate navigation link 530 of FIG. 5 , or a navigation linkprovided on another of the user interface screens described in thisExample.

The screen 600 can include a user interface control 608 that allows auser to select whether to use a guided analysis, such as according to adisclosed technology, or whether, for example, the user wishes tomanually define a machine learning task. In a particular example, if auser selects to exit a guided mode from the screen 600, the screen 600may continue to be used, but guidance or constraints that would beprovided in guided mode are not displayed, or are not enforced.Similarly, exiting guided mode from the screen 600 may cause guidance orconstraints to not be displayed or enforced if the user navigates toanother of the screens 700, 800, 900, or if the user returns to thescreen 500.

A user interface control 612 can allow a user to select a data set foruse. For example, a browse control 614 can allow a user to select orbrowse a repository of data sets. Window 616 shows a list of availabledata sets (e.g., in a repository currently being browsed). In somecases, a guidance framework, such as the guidance framework 108 of FIG.1 , can filter data sets so that only data sets that comply with aselected machine learning model, including based on any prior specificconfigurations made to an instance of such model. In other cases, agreater number of data sets can be shown, including non-compatible datasets. However, indications or warnings can be provided if a data set isnot compatible with an instance of a machine learning model that isbeing configured. As shown, for example, data set 620 has beendetermined not to comply with the instance of the machine learning modelbeing configured by the user. A warning indicator 624 is displayed, anda window 626 provides an explanation for the warning.

Typically, when a guided mode is selected, a user is not permitted toselect data sets that a guidance framework determines to beincompatible. In other cases, warnings are provided, but the user is notprohibited from selecting what may be incompatible data sets (eventhough a machine learning task may later fail). Or, warnings can beclassified as critical or non-critical, where a non-critical warning canindicate a potential problem (e.g., a data set not having enough data toprovide reliable results), but does not prevent selection of the dataset, while a critical warning, such as data being in an incompatibleformat, can result in the flagged data set not being selectable.

The screen 600 is shown as including a model visualization window orpanel 630. Once a machine learning model has been selected, a visualmodel representation 632 can be displayed in the window 630. The visualmodel representation 632 can allow a user to view all elements of themachine learning model even if a particular user interface screen onlyrelates to a particular element of the model. As discussed above,providing this contextual information can facilitate modelconfiguration, reduce user frustration, and help train users so thatthey may consider using more complex models (e.g., by classifyingthemselves as a more sophisticated user, so that they will be presentedwith more options).

The visual model representation 632 can identify one or more machinelearning model components that are currently active, such as if thescreen 600 allows a user to configure such components. For example, asthe screen 600 relates to data set selection, the data set component 634a of the visual representation 632 can be highlighted or otherwisedisplayed in a visually distinguished manner. In some cases, a user canmove to different workflow steps by selecting different components 634of the visual representation 632. Or, a user may be provided withdetails regarding a particular machine learning model component 634 whenthe model is selected (e.g., if an algorithm component is selected, thename of the algorithm and algorithm settings can be displayed).

A preview panel or window 640 can display information related to datasets, such as data sets available for selection using the window 616. Asshown, the panel 640 includes data set representations 642, where arepresentation includes a name or identifier 644 of the data set and avisual depiction or summary of data in the data set, such as a bargraph, line graph, or other type of data visualization.

The screen 600 can include navigation options 650, which can beimplemented at least generally as described for the navigation links 530of FIG. 5 .

The screen 700 can be displayed to the user once the screen 600 has beencompleted, when the user selects an appropriate navigation option fromanother screen, or at least after the user has selected a data set usingthe user interface control 612 of FIG. 6 . Elements of the screen 600that are reused in the screen 700 retain the reference numbers used inthe description of FIG. 6 .

Once a data set is selected, a user may be presented with additionaloptions for the data set. In some cases, these additional options can berepresented as sub-objects, sub-entities, or sub-components in a machinelearning model. In other cases, the additional options may be associatedwith a more primary object, entity, or component, or at least notvisually distinguished as being associated with a sub-entity orsub-component. The additional options can include options to filter orotherwise process or select data in a data set. For example, the screen700 is shown as including a first filter field 706, which has been setto select data elements (e.g., database records) having a value ofgreater than 100 in a first column. A second filter field 708 is shownas restricting data elements to those associated with a particular date.

The additional options can include options specific to a particularalgorithm, use case, or problem type. For example, for machine learningtechniques that require training data, a user can use a slider controlelement 712 to select a split of data in the data set, where a firstportion can be used for training data and a second portion used forvalidation. Or, the second portion may be used for analysis orclassification (e.g., the second portion of data may be analyzed usingthe model trained using the first portion of the data).

Note that, as a data set has been selected, the data set representation642 a for the selected data set is displayed in a visuallydistinguishable manner, such as being highlighted or being displayed ina different color or style than representations 642 of unselected datasets.

The screen 800 of FIG. 8 can allow a user to select, and optionallyconfigure elements of, an algorithm to be used in the instance of themachine learning model being configured. The screen 800 can include auser interface control 808 that allows a user to select whether to use aguided analysis, such as according to a disclosed technology, orwhether, for example, the user wishes to manually define a machinelearning task. In a particular example, if a user selects to exit aguided mode from the screen 800, the screen 800 may continue to be used,but guidance or constraints that would be provided in guided mode arenot displayed, or are not enforced. Similarly, exiting guided mode fromthe screen 800 may cause guidance or constraints to not be displayed orenforced if the user navigates to another of the screens 500, 600, 700,or 900.

The user can select an algorithm in field 812. The field 812 can beassociated with a browse control element 816, which can display to auser a list of available algorithms. The list of available algorithmscan be constrained, such as based on a level (e.g., an experience orcomplexity level) selected by a user, such as in the screen 500, aparticular problem or use case selected by the user, or other factors.For example, a data set selected using the screens 600/700 can, at leastin some cases, limit algorithms that can be selected for the field 812to those that are consistent with the selected data set.

Once an algorithm is selected, additional user interface controls can bedisplayed that either allow a user to configure various settings for thealgorithm, or which display settings to the user, even if the user maynot change the value assigned to the setting, such as if the setting isonly changeable for more experienced users (or otherwise associated witha level that indicates additional complexity is to be provided in amachine learning model). In the particular example shown, a Naive BayesClassifier has been selected in field 812 as the algorithm to be used.Based on that selection, fields 820, 822, 824 for three parameters ofthe Naive Bayes Classifier are shown. Field 820 can relate to a way ofconverting textual features to numerical input that can be processed bythe Naive Bayes algorithm, and indicates that word frequency is to beused. Field 822 relates to a smoothing technique to be used, and showsthat Laplace smoothing has been selected. Field 824 denotes whetherprobabilities for new cases should be affected by probabilitiesdetermined by cases analyzed to date. Field 824 can be highlighted orotherwise visually distinguished to indicate that is not available forselection by a user (at least given a current level set for or by theuser).

The screen 800 can include a model visualization panel 830, which can beat least generally similar to the model visualization panel 630 of FIG.6 . A visual representation 832 of the machine learning model incudes avisual indication 834, such as a check mark, for a data set component836, indicating that this component has been configured by the user. Arepresentation 838 of the algorithm component of the machine learningmodel is highlighted or otherwise visually distinguished to indicatethat it is active/associated with the screen 800.

The screen 800 can include a preview panel or window 840, which can beat least similar to the preview panel or window 630 of FIG. 6 . Previewpanels or windows of the user interface screens of this Example aregenerally context sensitive, as is a portion of the screens thatprovides configuration options (e.g., the portion with the fields 812,820, 822, 824). Accordingly, the preview window 840 provides additionalinformation regarding algorithms that are available for selection in thefield 812. A representation 844 of an algorithm displayed in the previewwindow 840 can provide information such as pros and cons of using thealgorithm (for example, easy to understand output, high computingresource use), complexity of the algorithm (which could be computationalcomplexity, interpretation complexity, complexity of configurationsettings, or other measures of complexity), a relative description ofcomputing resources used by the algorithm, any specific datarequirements (e.g., numeric data, a minimum number of data points,labelled data, a training data set, etc.), and identifiers of anysetting associated with the algorithm, or at least settings that areavailable for user configuration.

The screen 800 can highlight or otherwise distinguish a representation844 (e.g., 844 a, as shown), for a currently selected algorithm. Thescreen can include navigation options 850, which can be at least similarto the navigation links 530 of FIG. 5 .

The screen 900 of FIG. 9 can be used to start, or view the status of, amachine learning task configured at least in part using the screens500-800. The screen 900 can include an identifier 912 of the algorithmbeing used, such as k-means clustering, as shown. The screen 900 canalso be used to select output parameters that will be used in providingor processing results of the algorithm. For example, the k-meansclustering algorithm may provide class label output 916 as a defaultoption. A user optionally may select to include the F1 score in theoutput results by selecting user interface control 918. The F1 score canbe based on the precision and recall factors, which can be associatedwith a user interface control 922. As shown, the control 922 is notavailable for selection by the user, such as because this output settingis associated with a user/complexity level that is higher/more complexthan a level selected by or for a current user.

The screen 900 can also be used to provide information regarding amachine learning task that is being executed, or to control theexecution of the task. For example, indicators 930, 932 can provideindicators of CPU and memory use, respectively. Controls 934, 936 can beused to restart or pause/resume a task, respectively. Controls 938, 940can be used, respectively, to increase or decrease a prediction orconfidence interval to be used with results of the algorithm.

The screen 900 can provide additional details regarding the execution ofa machine learning task. For example, a progress bar 944 can indicateprogress in completing the machine learning task. Bars 946, 948 can beused, respectively, to provide information regarding the accuracy anderror associated with the algorithm at the completion degree indicatedin the progress bar 944.

The screen 900 can include a model visualization panel 950, which can beat least generally similar to the model visualization panel 630 of FIG.6 . A representation 952 of the output component of a machine learningmodel can be highlighted or otherwise visually distinguished to indicatethat it is active/associated with the screen 900.

A preview window or panel 960 can provide details regarding algorithmoutput settings or execution. As shown, the preview window 960 includesa representation 962 of statistics calculated based on data in the dataset, and a representation 964 of the output of the algorithm based onthe level of progress indicated by the progress bar 944. The screen 900can include navigation options 970, which can be at least generallysimilar to the navigation links 530 of FIG. 5 .

Example 7 Example Implementations

FIG. 10 is a flowchart of a method 1000 for configuring and executing acomputer-implemented process defined by a data model. The method 1000can be carried out in the computing environment 100 of FIG. 1 . At 1008,a data model is received that includes a plurality of data objects. Eachdata object has an object type. A first display is rendered at 1012,requesting selection of a first value for a first data object having afirst data object type. The first display also provides a schematicdiagram of the plurality of data objects in the data model. The firstvalue is assigned to the first data object at 1016. At 1020, a seconddisplay is rendered requesting selection of a second value for a seconddata object having a second data object type. The second display alsoprovides the schematic diagram. The second value is assigned to thesecond data object at 1024. At 1028, a computer-implemented processdefined by the data model is executed using the first value and thesecond value to provide execution results. The execution results aredisplayed to a user at 1032.

FIG. 11 is a flowchart of a method 1100 for configuring and executing acomputer-implemented process defined by a data model. The method 1100can be carried out in the computing environment 100 of FIG. 1 . At 1108,a selection of a data model type is received. The data model type isdefined by a sequenced or hierarchical relationship of a plurality ofdata objects of a plurality of different data object types. A data modelcorresponding to the data model type is retrieved at 1112. At 1116, afirst display is rendered requesting at least a first value to beassigned to a first data object of the plurality of data objects. Thefirst data object has a first type. The first display depicts thesequenced or hierarchical relationship of the plurality of data objects.A second display is rendered at 1120. The second display requests atleast a second value to be assigned to a second data object of theplurality of data objects. The second data object has a second type. Thesecond display depicts the sequenced or hierarchical relationship of theplurality of data objects. At 1124, a computer-implemented processdefined by the data model is executed using the at least a first valueand the at least a second value. Execution results are displayed at1128.

FIG. 12 is a flowchart of a method 1200 for configuring and executing amachine learning model. The method 1200 can be carried out in thecomputing environment 100 of FIG. 1 . A selection of a task level isreceived at 1208. At 1212, a plurality of machine learning models,stored in a repository, are determined that satisfy the selected tasklevel. A selection of a machine learning problem type or a use case isreceived at 1216. At 1220, at least one of the plurality of machinelearning models that satisfies the problem type or the use case isdetermined. A representation of the at least one machine learning modelis displayed at 1224. The representation illustrates a sequenced orhierarchical relationship between a plurality of components of the atleast one machine learning model. At least a first component of theplurality of components represents a data set to be processed and atleast a second component of the plurality of components represents atleast one algorithm to be used to process the data set. At 1228, inputspecifying the data set is received. The specified data set is processedusing an algorithm of the subset at 1232. Execution results are providedat 1236.

Example 8 Computing Systems

FIG. 13 depicts a generalized example of a suitable computing system1300 in which the described innovations may be implemented. Thecomputing system 1300 is not intended to suggest any limitation as toscope of use or functionality of the present disclosure, as theinnovations may be implemented in diverse general-purpose orspecial-purpose computing systems.

With reference to FIG. 13 , the computing system 1300 includes one ormore processing units 1310, 1315 and memory 1320, 1325. In FIG. 13 ,this basic configuration 1330 is included within a dashed line. Theprocessing units 1310, 1315 execute computer-executable instructions,such as for implementing components of the environment 100 of FIG. 1 ,including as described in Examples 1-7. A processing unit can be ageneral-purpose central processing unit (CPU), processor in anapplication-specific integrated circuit (ASIC), or any other type ofprocessor. In a multi-processing system, multiple processing unitsexecute computer-executable instructions to increase processing power.For example, FIG. 13 shows a central processing unit 1310 as well as agraphics processing unit or co-processing unit 1315. The tangible memory1320, 1325 may be volatile memory (e.g., registers, cache, RAM),non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or somecombination of the two, accessible by the processing unit(s) 1310, 1315.The memory 1320, 1325 stores software 1380 implementing one or moreinnovations described herein, in the form of computer-executableinstructions suitable for execution by the processing unit(s) 1310,1315.

A computing system 1300 may have additional features. For example, thecomputing system 1300 includes storage 1340, one or more input devices1350, one or more output devices 1360, and one or more communicationconnections 1370. An interconnection mechanism (not shown) such as abus, controller, or network interconnects the components of thecomputing system 1300. Typically, operating system software (not shown)provides an operating environment for other software executing in thecomputing system 1300, and coordinates activities of the components ofthe computing system 1300.

The tangible storage 1340 may be removable or non-removable, andincludes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, orany other medium which can be used to store information in anon-transitory way and which can be accessed within the computing system1300. The storage 1340 stores instructions for the software 1380implementing one or more innovations described herein.

The input device(s) 1350 may be a touch input device such as a keyboard,mouse, pen, or trackball, a voice input device, a scanning device, oranother device that provides input to the computing system 1300. Theoutput device(s) 1360 may be a display, printer, speaker, CD-writer, oranother device that provides output from the computing system 1300.

The communication connection(s) 1370 enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio or video input or output, or other data in a modulated datasignal. A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia can use an electrical, optical, RF, or other carrier.

The innovations can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing system on a target real orvirtual processor. Generally, program modules or components includeroutines, programs, libraries, objects, classes, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. The functionality of the program modules may becombined or split between program modules as desired in variousembodiments. Computer-executable instructions for program modules may beexecuted within a local or distributed computing system.

The terms “system” and “device” are used interchangeably herein. Unlessthe context clearly indicates otherwise, neither term implies anylimitation on a type of computing system or computing device. Ingeneral, a computing system or computing device can be local ordistributed, and can include any combination of special-purpose hardwareand/or general-purpose hardware with software implementing thefunctionality described herein.

In various examples described herein, a module (e.g., component orengine) can be “coded” to perform certain operations or provide certainfunctionality, indicating that computer-executable instructions for themodule can be executed to perform such operations, cause such operationsto be performed, or to otherwise provide such functionality. Althoughfunctionality described with respect to a software component, module, orengine can be carried out as a discrete software unit (e.g., program,function, class method), it need not be implemented as a discrete unit.That is, the functionality can be incorporated into a larger or moregeneral purpose program, such as one or more lines of code in a largeror general purpose program.

For the sake of presentation, the detailed description uses terms like“determine” and “use” to describe computer operations in a computingsystem. These terms are high-level abstractions for operations performedby a computer, and should not be confused with acts performed by a humanbeing. The actual computer operations corresponding to these terms varydepending on implementation.

Example 9 Cloud Computing Environment

FIG. 14 depicts an example cloud computing environment 1400 in which thedescribed technologies can be implemented. The cloud computingenvironment 1400 comprises cloud computing services 1410. The cloudcomputing services 1410 can comprise various types of cloud computingresources, such as computer servers, data storage repositories,networking resources, etc. The cloud computing services 1410 can becentrally located (e.g., provided by a data center of a business ororganization) or distributed (e.g., provided by various computingresources located at different locations, such as different data centersand/or located in different cities or countries).

The cloud computing services 1410 are utilized by various types ofcomputing devices (e.g., client computing devices), such as computingdevices 1420, 1422, and 1424. For example, the computing devices (e.g.,1420, 1422, and 1424) can be computers (e.g., desktop or laptopcomputers), mobile devices (e.g., tablet computers or smart phones), orother types of computing devices. For example, the computing devices(e.g., 1420, 1422, and 1424) can utilize the cloud computing services1410 to perform computing operators (e.g., data processing, datastorage, and the like).

Example 10 Implementations

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed methods can be used in conjunction with other methods.

Any of the disclosed methods can be implemented as computer-executableinstructions or a computer program product stored on one or morecomputer-readable storage media, such as tangible, non-transitorycomputer-readable storage media, and executed on a computing device(e.g., any available computing device, including smart phones or othermobile devices that include computing hardware). Tangiblecomputer-readable storage media are any available tangible media thatcan be accessed within a computing environment (e.g., one or moreoptical media discs such as DVD or CD, volatile memory components (suchas DRAM or SRAM), or nonvolatile memory components (such as flash memoryor hard drives)). By way of example, and with reference to FIG. 13 ,computer-readable storage media include memory 1320 and 1325, andstorage 1340. The term computer-readable storage media does not includesignals and carrier waves. In addition, the term computer-readablestorage media does not include communication connections (e.g., 1370).

Any of the computer-executable instructions for implementing thedisclosed techniques as well as any data created and used duringimplementation of the disclosed embodiments can be stored on one or morecomputer-readable storage media. The computer-executable instructionscan be part of, for example, a dedicated software application or asoftware application that is accessed or downloaded via a web browser orother software application (such as a remote computing application).Such software can be executed, for example, on a single local computer(e.g., any suitable commercially available computer) or in a networkenvironment (e.g., via the Internet, a wide-area network, a local-areanetwork, a client-server network (such as a cloud computing network), orother such network) using one or more network computers.

For clarity, only certain selected aspects of the software-basedimplementations are described. Other details that are well known in theart are omitted. For example, it should be understood that the disclosedtechnology is not limited to any specific computer language or program.For instance, the disclosed technology can be implemented by softwarewritten in C, C++, C#, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL,XCode, GO, Adobe Flash, or any other suitable programming language, or,in some examples, markup languages such as html or XML, or combinationsof suitable programming languages and markup languages. Likewise, thedisclosed technology is not limited to any particular computer or typeof hardware. Certain details of suitable computers and hardware are wellknown and need not be set forth in detail in this disclosure.

Furthermore, any of the software-based embodiments (comprising, forexample, computer-executable instructions for causing a computer toperform any of the disclosed methods) can be uploaded, downloaded, orremotely accessed through a suitable communication means. Such suitablecommunication means include, for example, the Internet, the World WideWeb, an intranet, software applications, cable (including fiber opticcable), magnetic communications, electromagnetic communications(including RF, microwave, and infrared communications), electroniccommunications, or other such communication means.

The disclosed methods, apparatus, and systems should not be construed aslimiting in any way. Instead, the present disclosure is directed towardall novel and nonobvious features and aspects of the various disclosedembodiments, alone and in various combinations and sub combinations withone another. The disclosed methods, apparatus, and systems are notlimited to any specific aspect or feature or combination thereof, nor dothe disclosed embodiments require that any one or more specificadvantages be present or problems be solved.

The technologies from any example can be combined with the technologiesdescribed in any one or more of the other examples. In view of the manypossible embodiments to which the principles of the disclosed technologymay be applied, it should be recognized that the illustrated embodimentsare examples of the disclosed technology and should not be taken as alimitation on the scope of the disclosed technology. Rather, the scopeof the disclosed technology includes what is covered by the scope andspirit of the following claims.

What is claimed is:
 1. A computing system comprising: at least onememory; one or more hardware processing units coupled to the at leastone memory; and one or more non-transitory computer readable storagemedia storing instructions that, when executed, cause the computingsystem to perform operations comprising: receiving a selection of afirst data model type from a plurality of data model types, where datamodels according to a data model type comprise a plurality of dataobjects, the plurality of data objects being of multiple data objecttypes, and where a given data model defines a hierarchy or sequencebetween its data objects and one or both of a number and a type of dataobjects, or the hierarchy or sequence between data objects, differsbetween at least a portion of the data models; rendering a first displayof a first data model according to the first data model type, the firstdisplay (1) requesting selection of a first value for a first dataobject having a first object type, and (2) providing a schematic diagramof the plurality of data objects in the first data model, the schematicdiagram being defined at least in part by the hierarchy or sequencebetween data objects in the first data model type, wherein the firstdata model is associated with a first problem type and a first userlevel and is selected from the plurality of data model types, theplurality of data model types having at least a second data modelassociated with the first problem type and associated with a second userlevel, and wherein the second data object is of the first object typeand has at least one sub-object type that is configurable by a user andis not configurable by the user in the first data model type; assigningthe first value to the first data object; rendering a second display (1)requesting selection of a second value for a second data object having asecond object type, and (2) providing the schematic diagram; assigningthe second value to the second data object; executing acomputer-implemented process defined by the first data model using thefirst value and the second value to provide execution results; anddisplaying the execution results to a user.
 2. The computing system ofclaim 1, wherein the first data model represents a machine learningprocess and the first object type represents a data set.
 3. Thecomputing system of claim 2, wherein the second object type represents amachine learning algorithm.
 4. The computing system of claim 1, whereinthe first data model represents a machine learning process and the firstvalue is constrained to a first set of values set for the machinelearning process.
 5. The computing system of claim 4, wherein the firstdata model is selected from a plurality of data models and the machinelearning process is a first machine learning process, third data modelof the plurality of data models representing a second machine learningprocess and a third value for a first data object of the third datamodel is constrained to a second set of values set for the secondmachine learning process, wherein the second set of values has at leastone member that differs from members of the first set values.
 6. Thecomputing system of claim 1, the processing further comprising:receiving a selection of a user level to provide a selected user level;determining a first portion of the plurality of data model typesavailable for selection by users having the selected user level;displaying at least a second portion of the first portion of theplurality of data model types to a user; receiving a selection of aproblem type; determining a third portion of the plurality of data modeltypes having identifiers indicating they are associated with the problemtype; and displaying at least a fourth portion of the third portion ofthe plurality of data model types to the user.
 7. A method, implementedin a computing system comprising a memory and one or more hardwareprocessors coupled to the memory, comprising: receiving a selection of afirst data model type of a plurality of data model types, wherein datamodels of the first data model type define a sequenced or hierarchicalrelationship of a plurality of data objects of a plurality of differentdata object types, the sequenced or hierarchical relationship differingbetween one or both of data model types or data models of a given datamodel type; retrieving a first data model corresponding to the firstdata model type, wherein the first data model is associated with a firstproblem type and a first user level and is selected from the pluralityof data model types, the plurality of data model types having at least asecond data model associated with the first problem type and associatedwith a second user level, and wherein the second data object is of thefirst object type and has at least one sub-object type that isconfigurable by a user and is not configurable by the user in the firstdata model type; rendering a first display (1) requesting at least afirst value to be assigned to a first data object of the plurality ofdata objects, the first data object having a first type, and (2)depicting the sequenced or hierarchical relationship of the plurality ofdata objects; rendering a second display (1) requesting at least asecond value to be assigned to a second data object of the plurality ofdata objects, the second data object having a second type, and (2)depicting the sequenced or hierarchical relationship of the plurality ofdata objects, including depicting a sequenced or hierarchicalrelationship of the first data object and the second data object;executing a computer-implemented process defined by the first data modelusing the at least a first value and the at least a second value; anddisplaying execution results.
 8. The method of claim 7, wherein the atleast a second value is selectable from a first set prior to receiving aselection of a value to be assigned as the at least a first value andthe at least a second value is selectable from a second set thereafter,wherein the second set is a proper subset of the first set.
 9. Themethod of claim 7, wherein the first data model type is associated witha machine learning task, the at least a first value identifies a dataset to be used in the machine learning task, and the at least a secondvalue identifies an algorithm to operate on at least a portion of thedata set.
 10. The method of claim 7, wherein the selection of a datamodel type comprises a selection of a task level.
 11. The method ofclaim 10, wherein the selection of a data model type further comprises atask purpose.
 12. The method of claim 11, wherein the first data modeltype is associated with a machine learning task.
 13. One or moretangible computer-readable storage media storing computer-executableinstructions that, when executed by a computing system, cause thecomputing system to perform processing comprising: receiving a selectionof a task level; determining a plurality of machine learning modelsstored in a repository that match the selected task level; receiving aselection of a machine learning problem type or a use case; determiningat least one of the plurality of machine learning models that match themachine learning problem type or the use case, wherein machine learningmodels of the plurality of machine learning models comprise a pluralityof components of a plurality of component types, where one or both of anumber of components or component types differ between at least aportion of the machine learning models of the plurality of machinelearning models; displaying a representation of at least one machinelearning model of the determined at least one of the plurality ofmachine learning models, the representation illustrating a sequenced orhierarchical relationship between a plurality of components of the atleast one machine learning model, wherein at least a first component ofthe plurality of components represents a data set to be processed and atleast a second component of the plurality of components represents atleast one algorithm to be used to process the data set; receiving inputspecifying the data set; processing the specified data set using the atleast one algorithm; and providing executing results.
 14. The one ormore tangible computer-readable storage media of claim 13, wherein themachine learning problem type is selected from a set comprisingclassification or clustering.
 15. The one or more tangiblecomputer-readable storage media of claim 13, wherein machine learningmodels of the plurality of machine learning models specify respectivesets of one or more algorithms applicable to a respective machinelearning model.
 16. The one or more tangible computer-readable storagemedia of claim 15, wherein the at least one machine learning modelcomprises at least one component that constrains a set of a plurality ofalgorithms useable as the at least one algorithm when a value isassigned to the at least one component, providing a constrained set. 17.The one or more tangible computer-readable storage media of claim 16,the processing further comprising: receiving user input selecting analgorithm available in the constrained set.
 18. The one or more tangiblecomputer-readable storage media of claim 17, the processing furthercomprising: displaying the representation during the receiving userinput selecting an algorithm.
 19. The one or more tangiblecomputer-readable storage media of claim 13, further comprising:displaying the representation during the receiving input specifying thedata set.
 20. A computing system comprising: at least one memory; one ormore hardware processing units coupled to the at least one memory; andone or more non-transitory computer readable storage media storinginstructions that, when executed, cause the computing system to performoperations comprising: receiving a selection of a user level to providea selected user level; determining a first portion of a plurality ofdata model types available for selection by users having the selecteduser level; displaying at least a second portion of the first portion ofthe plurality of data model types to a user; receiving a selection of aproblem type; determining a third portion of the plurality of data modeltypes having identifiers indicating they are associated with the problemtype; and displaying at least a fourth portion of the third portion ofthe plurality of data model types to the user; receiving a selection ofa first data model type from the plurality of data model types, wheredata models according to a data model type comprise a plurality of dataobjects, the plurality of data objects being of multiple data objecttypes, and where a given data model defines a hierarchy or sequencebetween its data objects and one or both of a number and a type of dataobjects, or the hierarchy or sequence between data objects, differsbetween at least a portion of the data models; rendering a first displayof a data model according to the first data model type, the firstdisplay (1) requesting selection of a first value for a first dataobject having a first object type, and (2) providing a schematic diagramof the plurality of data objects in the data model, the schematicdiagram being defined at least in part by the hierarchy or sequencebetween data objects in the first data model type; assigning the firstvalue to the first data object; rendering a second display (1)requesting selection of a second value for a second data object having asecond object type, and (2) providing the schematic diagram; assigningthe second value to the second data object; executing acomputer-implemented process defined by the data model using the firstvalue and the second value to provide execution results; and displayingthe execution results to a user.